{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "5e8ae3d7-3e2e-4755-a0b6-709ef4180719",
   "metadata": {},
   "source": [
    "Copyright (c) MONAI Consortium  \n",
    "Licensed under the Apache License, Version 2.0 (the \"License\");  \n",
    "you may not use this file except in compliance with the License.  \n",
    "You may obtain a copy of the License at  \n",
    "&nbsp;&nbsp;&nbsp;&nbsp;http://www.apache.org/licenses/LICENSE-2.0  \n",
    "Unless required by applicable law or agreed to in writing, software  \n",
    "distributed under the License is distributed on an \"AS IS\" BASIS,  \n",
    "WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.  \n",
    "See the License for the specific language governing permissions and  \n",
    "limitations under the License."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "191c5d77-8ae5-49ab-be22-45f5ba41641f",
   "metadata": {},
   "source": [
    "## Setup environment"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "886952c4-0be4-459d-9c53-b81b29199c76",
   "metadata": {},
   "outputs": [],
   "source": [
    "!python -c \"import monai\" || pip install -q \"monai-weekly[ignite,pyyaml]\""
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a20e1274-0a27-4e37-95d7-fb813243c34c",
   "metadata": {},
   "source": [
    "## Setup imports"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "b1144d87-ec2f-4b9b-907a-16ea2da279c4",
   "metadata": {},
   "outputs": [],
   "source": [
    "from monai.config import print_config\n",
    "\n",
    "print_config()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c572d8b6-3dca-4487-80ad-928090b3e8ab",
   "metadata": {},
   "source": [
    "# MedNIST Classification Bundle\n",
    "\n",
    "In this tutorial we'll describe how to create a bundle for a classification network. This will include how to train and apply the network on the command line. MedNIST will be used as the dataset with the bundle based off the [MONAI 101 notebook](https://github.com/Project-MONAI/tutorials/blob/main/2d_classification/monai_101.ipynb).\n",
    "\n",
    "The dataset is kindly made available by Dr. Bradley J. Erickson M.D., Ph.D. (Department of Radiology, Mayo Clinic) under the Creative Commons CC BY-SA 4.0 license. If you use the MedNIST dataset, please acknowledge the source of the MedNIST dataset: the repository https://github.com/Project-MONAI/MONAI/ or the MedNIST tutorial for image classification https://github.com/Project-MONAI/MONAI/blob/master/examples/notebooks/mednist_tutorial.ipynb.\n",
    "\n",
    "This work is licensed under the Creative Commons Attribution-ShareAlike 4.0 International License. To view a copy of this license, visit http://creativecommons.org/licenses/by-sa/4.0/.\n",
    "\n",
    "First we'll consider a condensed version of the code from that notebook and go step-by-step how best to represent this as a bundle:\n",
    "\n",
    "```python\n",
    "import os\n",
    "\n",
    "import monai.transforms as mt\n",
    "import torch\n",
    "from monai.apps import MedNISTDataset\n",
    "from monai.data import DataLoader\n",
    "from monai.engines import SupervisedTrainer\n",
    "from monai.inferers import SimpleInferer\n",
    "from monai.networks import eval_mode\n",
    "from monai.networks.nets import densenet121\n",
    "\n",
    "root_dir = os.environ.get(\"ROOTDIR\", \".\")\n",
    "\n",
    "max_epochs = 25\n",
    "device = torch.device(\"cuda:0\")\n",
    "net = densenet121(spatial_dims=2, in_channels=1, out_channels=6).to(device)\n",
    "\n",
    "transform = mt.Compose([\n",
    "        mt.LoadImaged(keys=\"image\", image_only=True),\n",
    "        mt.EnsureChannelFirstd(keys=\"image\"),\n",
    "        mt.ScaleIntensityd(keys=\"image\"),\n",
    "])\n",
    "\n",
    "dataset = MedNISTDataset(root_dir=root_dir, transform=transform, section=\"training\", download=True)\n",
    "\n",
    "train_dl = DataLoader(dataset, batch_size=512, shuffle=True, num_workers=4)\n",
    "\n",
    "trainer = SupervisedTrainer(\n",
    "    device=device,\n",
    "    max_epochs=max_epochs,\n",
    "    train_data_loader=train_dl,\n",
    "    network=net,\n",
    "    optimizer=torch.optim.Adam(net.parameters(), lr=1e-5),\n",
    "    loss_function=torch.nn.CrossEntropyLoss(),\n",
    "    inferer=SimpleInferer(),\n",
    ")\n",
    "\n",
    "trainer.run()\n",
    "\n",
    "torch.jit.script(net).save(\"mednist.ts\")\n",
    "\n",
    "class_names = (\"AbdomenCT\", \"BreastMRI\", \"CXR\", \"ChestCT\", \"Hand\", \"HeadCT\")\n",
    "testdata = MedNISTDataset(root_dir=root_dir, transform=transform, section=\"test\", runtime_cache=True)\n",
    "\n",
    "max_items_to_print = 10\n",
    "eval_dl = DataLoader(testdata[:max_items_to_print], batch_size=1, num_workers=0)\n",
    "with eval_mode(net):\n",
    "    for item in eval_dl:\n",
    "        result = net(item[\"image\"].to(device))\n",
    "        prob = result.detach().to(\"cpu\")[0]\n",
    "        pred = class_names[prob.argmax()]\n",
    "        gt = item[\"class_name\"][0]\n",
    "        print(f\"Prediction: {pred}. Ground-truth: {gt}\")\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1a18d5cd-6338-4b41-87fd-4e119723bfee",
   "metadata": {},
   "source": [
    "You can run this cell or save it to a file and run it on the command line. A `DenseNet` based network will be trained to classify MedNIST images into one of six categories. Mostly this script uses Ignite-based classes such as `SupervisedTrainer` which is great for converting into a bundle. Let's start by initialising a bundle directory structure:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "eb9dc6ec-13da-4a37-8afa-28e2766b9343",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "/usr/bin/tree\n",
      "\u001b[01;34mMedNISTClassifier\u001b[00m\n",
      "├── \u001b[01;34mconfigs\u001b[00m\n",
      "│   ├── inference.json\n",
      "│   └── metadata.json\n",
      "├── \u001b[01;34mdocs\u001b[00m\n",
      "│   └── README.md\n",
      "├── LICENSE\n",
      "└── \u001b[01;34mmodels\u001b[00m\n",
      "\n",
      "3 directories, 4 files\n"
     ]
    }
   ],
   "source": [
    "%%bash\n",
    "\n",
    "python -m monai.bundle init_bundle MedNISTClassifier\n",
    "which tree && tree MedNISTClassifier || true"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5888c9bd-5022-40b5-9dec-84d9f737f868",
   "metadata": {},
   "source": [
    "## Metadata\n",
    "\n",
    "We'll first replace the `metadata.json` file with our description of what the network will do:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "b29f053b-cf16-4ffc-bbe7-d9433fdfa872",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Overwriting MedNISTClassifier/configs/metadata.json\n"
     ]
    }
   ],
   "source": [
    "%%writefile MedNISTClassifier/configs/metadata.json\n",
    "\n",
    "{\n",
    "    \"version\": \"0.0.1\",\n",
    "    \"changelog\": {\n",
    "        \"0.0.1\": \"Initial version\"\n",
    "    },\n",
    "    \"monai_version\": \"1.2.0\",\n",
    "    \"pytorch_version\": \"2.0.0\",\n",
    "    \"numpy_version\": \"1.23.5\",\n",
    "    \"required_packages_version\": {},\n",
    "    \"name\": \"MedNISTClassifier\",\n",
    "    \"task\": \"MedNIST Classification Network\",\n",
    "    \"description\": \"This is a demo network for classifying MedNIST images by type/modality\",\n",
    "    \"authors\": \"Your Name Here\",\n",
    "    \"copyright\": \"Copyright (c) Your Name Here\",\n",
    "    \"data_source\": \"MedNIST dataset kindly made available by Dr. Bradley J. Erickson M.D., Ph.D. (Department of Radiology, Mayo Clinic)\",\n",
    "    \"data_type\": \"jpeg\",\n",
    "    \"intended_use\": \"This is suitable for demonstration only\",\n",
    "    \"network_data_format\": {\n",
    "        \"inputs\": {\n",
    "            \"image\": {\n",
    "                \"type\": \"image\",\n",
    "                \"format\": \"magnitude\",\n",
    "                \"modality\": \"any\",\n",
    "                \"num_channels\": 1,\n",
    "                \"spatial_shape\": [64, 64],\n",
    "                \"dtype\": \"float32\",\n",
    "                \"value_range\": [0, 1],\n",
    "                \"is_patch_data\": false,\n",
    "                \"channel_def\": {\n",
    "                    \"0\": \"image\"\n",
    "                }\n",
    "            }\n",
    "        },\n",
    "        \"outputs\": {\n",
    "            \"pred\": {\n",
    "                \"type\": \"probabilities\",\n",
    "                \"format\": \"classes\",\n",
    "                \"num_channels\": 6,\n",
    "                \"spatial_shape\": [6],\n",
    "                \"dtype\": \"float32\",\n",
    "                \"value_range\": [0, 1],\n",
    "                \"is_patch_data\": false,\n",
    "                \"channel_def\": {\n",
    "                    \"0\": \"AbdomenCT\",\n",
    "                    \"1\": \"BreastMRI\",\n",
    "                    \"2\": \"CXR\",\n",
    "                    \"3\": \"ChestCT\",\n",
    "                    \"4\": \"Hand\",\n",
    "                    \"5\": \"HeadCT\"\n",
    "                }\n",
    "            }\n",
    "        }\n",
    "    }\n",
    "}"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3f208bf8-0c3a-4def-ab0f-6091cebcd532",
   "metadata": {},
   "source": [
    "This contains more information compared to the previous tutorial's file. For inputs the network, a tensor \"image\" is given as a 64x64 sized single-channel image. This is one of the MedNIST images whose modality varies but will have a value range of `[0, 1]` after rescaling in the transform pipeline. The channel definition states the meaning of each channel, this input has only one which is the greyscale image itself. For network outputs there is only one, \"pred\", representing the prediction of the network as a tensor of size 6. Each of the six values is a prediction of that class which is described in `channel_def`.\n",
    "\n",
    "## Common Definitions\n",
    "\n",
    "What we'll now do is construct the bundle configuration scripts to implement training, testing, and inference based off the original script file given above. Common definitions should be placed in a common file used with other scripts to reduce duplication. In our original script, the network definition and transform sequence will be used in multiple places so should go in this common file:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "d11681af-3210-4b2b-b7bd-8ad8dedfe230",
   "metadata": {
    "lines_to_next_cell": 2
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Writing MedNISTClassifier/configs/common.yaml\n"
     ]
    }
   ],
   "source": [
    "%%writefile MedNISTClassifier/configs/common.yaml\n",
    "# only need to import torch right now\n",
    "imports: \n",
    "- $import torch\n",
    "\n",
    "# define a default root directory value, this can overridden on the command line\n",
    "root_dir: \".\"\n",
    "\n",
    "# define a device for the network\n",
    "device: '$torch.device(''cuda:0'')'\n",
    "\n",
    "# store the class names for inference later\n",
    "class_names: [AbdomenCT, BreastMRI, CXR, ChestCT, Hand, HeadCT]\n",
    "\n",
    "# define the network separately, don't need to refer to MONAI types by name or import MONAI\n",
    "network_def:\n",
    "  _target_: densenet121\n",
    "  spatial_dims: 2\n",
    "  in_channels: 1\n",
    "  out_channels: 6\n",
    "\n",
    "# define the network to be the given definition moved to the device\n",
    "net: '$@network_def.to(@device)'\n",
    "\n",
    "# define a transform sequence by instantiating a Compose instance with a transform sequence\n",
    "transform:\n",
    "  _target_: Compose\n",
    "  transforms:\n",
    "  - _target_: LoadImaged\n",
    "    keys: 'image'\n",
    "    image_only: true\n",
    "  - _target_: EnsureChannelFirstd\n",
    "    keys: 'image'\n",
    "  - _target_: ScaleIntensityd\n",
    "    keys: 'image'"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "eaf81ea7-9ea3-4548-a32e-992f0b9bc0ab",
   "metadata": {},
   "source": [
    "Although this YAML is very different from the Python code it's defining essentially the same objects. Whether in YAML or JSON a bundle script defines an object instantiation as a dictionary containing the key `_target_` declaring the type to create, with other keys treated as arguments. A Python statement like `obj = ObjType(arg1=val1, arg2=val2)` is thus equivalent to \n",
    "\n",
    "```yaml\n",
    "obj:\n",
    "  _target_: ObjType\n",
    "  arg1: val1\n",
    "  arg2: val2\n",
    "```\n",
    "\n",
    "Note here that MONAI will import all its own symbols such that an explicit import statement is not needed nor is referring to types by fully qualified name, ie. `Compose` is adequate instead of `monai.transforms.Compose`. Definitions found in other packages or those in scripts associated with the bundle need to be referred to by the name they are imported as, eg. `torch.device` as show above.\n",
    "\n",
    "## Training\n",
    "\n",
    "For training we need a dataset, dataloader, and trainer object which will be used in the running \"program\":"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "4dfd052e-abe7-473a-bbf4-25674a3b20ea",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Writing MedNISTClassifier/configs/train.yaml\n"
     ]
    }
   ],
   "source": [
    "%%writefile MedNISTClassifier/configs/train.yaml\n",
    "\n",
    "max_epochs: 25\n",
    "\n",
    "dataset:\n",
    "  _target_: MedNISTDataset\n",
    "  root_dir: '@root_dir'\n",
    "  transform: '@transform'\n",
    "  section: training\n",
    "  download: true\n",
    "\n",
    "train_dl:\n",
    "  _target_: DataLoader\n",
    "  dataset: '@dataset'\n",
    "  batch_size: 512\n",
    "  shuffle: true\n",
    "  num_workers: 4\n",
    "\n",
    "trainer:\n",
    "  _target_: SupervisedTrainer\n",
    "  device: '@device'\n",
    "  max_epochs: '@max_epochs'\n",
    "  train_data_loader: '@train_dl'\n",
    "  network: '@net'\n",
    "  optimizer: \n",
    "    _target_: torch.optim.Adam\n",
    "    params: '$@net.parameters()'\n",
    "    lr: 0.00001  # learning rate set slow so that you can see network improvement over epochs\n",
    "  loss_function: \n",
    "    _target_: torch.nn.CrossEntropyLoss\n",
    "  inferer: \n",
    "    _target_: SimpleInferer\n",
    "\n",
    "train:\n",
    "- '$@trainer.run()'\n",
    "- '$torch.jit.script(@net).save(''model.ts'')'"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "de752181-80b1-4221-9e4a-315e5f7f22a6",
   "metadata": {},
   "source": [
    "There is a lot going on here but hopefully you see how this replicates the object definitions in the original source file. A few specific points:\n",
    "* References are made to objects defined in `common.yaml` such as `@root_dir`, so this file needs to be used in conjunction with this one.\n",
    "* A `max_epochs` hyperparameter is provided whose value you can change on the command line, eg. `--max_epochs 5`.\n",
    "* Definitions for the `optimizer`, `loss_function`, and `inferer` arguments of `trainer` are provided inline but it would be better practice to define these separately.\n",
    "* The learning rate is hard-coded as `1e-5`, it would again be better practice to define a separate `lr` hyperparameter, although it can be changed on the command line with `'--trainer#optimizer#lr' 0.001`.\n",
    "* The trained network is saved using Pytorch's `jit` module directly, better practice would be to provide a handler, such as `CheckpointSaver`, to the trainer or to an evaluator object, see other tutorial examples on how to do this. This was kept here to match the original example.\n",
    "\n",
    "Now the network can be trained by running the bundle:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "8357670d-fe69-4789-9b9a-77c0d8144b10",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "2024-09-18 08:16:57,243 - INFO - --- input summary of monai.bundle.scripts.run ---\n",
      "2024-09-18 08:16:57,243 - INFO - > config_file: ['./MedNISTClassifier/configs/common.yaml',\n",
      " './MedNISTClassifier/configs/train.yaml']\n",
      "2024-09-18 08:16:57,243 - INFO - > meta_file: './MedNISTClassifier/configs/metadata.json'\n",
      "2024-09-18 08:16:57,243 - INFO - > run_id: 'train'\n",
      "2024-09-18 08:16:57,243 - INFO - > max_epochs: 2\n",
      "2024-09-18 08:16:57,243 - INFO - ---\n",
      "\n",
      "\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "MedNIST.tar.gz: 59.0MB [00:03, 20.1MB/s]                              \n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "2024-09-18 08:17:00,407 - INFO - Downloaded: MedNIST.tar.gz\n",
      "2024-09-18 08:17:00,490 - INFO - Verified 'MedNIST.tar.gz', md5: 0bc7306e7427e00ad1c5526a6677552d.\n",
      "2024-09-18 08:17:00,490 - INFO - Writing into directory: ..\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Loading dataset: 100%|██████████| 47164/47164 [00:23<00:00, 2042.89it/s]\n",
      "INFO:ignite.engine.engine.SupervisedTrainer:Engine run resuming from iteration 0, epoch 0 until 2 epochs\n",
      "INFO:ignite.engine.engine.SupervisedTrainer:Epoch[1] Complete. Time taken: 00:00:16.954\n",
      "INFO:ignite.engine.engine.SupervisedTrainer:Epoch[2] Complete. Time taken: 00:00:16.772\n",
      "INFO:ignite.engine.engine.SupervisedTrainer:Engine run complete. Time taken: 00:00:34.112\n"
     ]
    }
   ],
   "source": [
    "%%bash\n",
    "\n",
    "BUNDLE=\"./MedNISTClassifier\"\n",
    "\n",
    "# run the bundle with epochs set to 2 for speed during testing, change this to get a better result\n",
    "python -m monai.bundle run train \\\n",
    "    --meta_file \"$BUNDLE/configs/metadata.json\" \\\n",
    "    --config_file \"['$BUNDLE/configs/common.yaml','$BUNDLE/configs/train.yaml']\" \\\n",
    "    --max_epochs 2\n",
    "\n",
    "# we'll use the trained network as the model object for this bundle\n",
    "mv model.ts $BUNDLE/models/model.ts\n",
    "\n",
    "# generate the saved dictionary file as well\n",
    "cd \"$BUNDLE/models\"\n",
    "python -c 'import torch; obj = torch.jit.load(\"model.ts\"); torch.save(obj.state_dict(), \"model.pt\")'"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "bbf58fac-b6d5-424d-9e98-1a30937f2116",
   "metadata": {},
   "source": [
    "As shown here the Torchscript object produced by the training is moved into the `models` directory of the bundle. The saved weight file is also produced by loading that file again and saving the state. Once again best practice would be to instead use `CheckpointSaver` to save weights in an output location before the final file is chosen for the bundle. \n",
    "\n",
    "## Evaluation\n",
    "\n",
    "To replicate the original example's code we'll need to put the evaluation loop code into a separate function and call it. The best practice would be to use an `Evaluator` class to do this with metric classes for assessing performance. Instead we'll stick close to the original code and demonstrate how to integrate your own code into a bundle.\n",
    "\n",
    "The first thing to do is put the evaluation loop into a function and store it in the `scripts` module within the bundle:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "fbad1a21-4dda-4b80-8e81-7d7e75307f9c",
   "metadata": {},
   "outputs": [],
   "source": [
    "!mkdir MedNISTClassifier/scripts"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "0c8725f7-f1cd-48f5-81a5-3f5a9ee03e9c",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Writing MedNISTClassifier/scripts/__init__.py\n"
     ]
    }
   ],
   "source": [
    "%%writefile MedNISTClassifier/scripts/__init__.py\n",
    "\n",
    "from monai.networks.utils import eval_mode\n",
    "\n",
    "def evaluate(net, dataloader, class_names, device):\n",
    "    with eval_mode(net):\n",
    "        for item in dataloader:\n",
    "            result = net(item[\"image\"].to(device))\n",
    "            prob = result.detach().to(\"cpu\")[0]\n",
    "            pred = class_names[prob.argmax()]\n",
    "            gt = item[\"class_name\"][0]\n",
    "            print(f\"Prediction: {pred}. Ground-truth: {gt}\")\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "abf40c4f-3349-4c40-9eef-811388ffd704",
   "metadata": {},
   "source": [
    "The `scripts` directory has to be a valid Python module so needs a `__init__.py` file, you can include other files and import them separately or import their members into this file. Here we defined `evaluate` to enclose the loop from the original script. This can then be called as part of a expression sequence \"program\":"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "id": "b4e1f99a-a68b-4aeb-bcf2-842f26609b52",
   "metadata": {
    "lines_to_next_cell": 2
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Writing MedNISTClassifier/configs/evaluate.yaml\n"
     ]
    }
   ],
   "source": [
    "%%writefile MedNISTClassifier/configs/evaluate.yaml\n",
    "\n",
    "imports: \n",
    "- $import scripts\n",
    "\n",
    "max_items_to_print: 10\n",
    "\n",
    "ckpt_file: \"\"\n",
    "\n",
    "testdata:\n",
    "  _target_: MedNISTDataset\n",
    "  root_dir: '@root_dir'\n",
    "  transform: '@transform'\n",
    "  section: test\n",
    "  download: false\n",
    "  runtime_cache: true\n",
    "\n",
    "eval_dl:\n",
    "  _target_: DataLoader\n",
    "  dataset: '$@testdata[:@max_items_to_print]'\n",
    "  batch_size: 1\n",
    "  num_workers: 0\n",
    "\n",
    "# loads the weights from the given file (which needs to be set on the command line) then calls \"evaluate\"\n",
    "evaluate:\n",
    "- '$@net.load_state_dict(torch.load(@ckpt_file, weights_only=True))'\n",
    "- '$scripts.evaluate(@net, @eval_dl, @class_names, @device)'\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "64bb2286-3107-49e9-8dbe-66fe1a2ae08c",
   "metadata": {},
   "source": [
    "Evaluation is then run on the command line, using \"evaluate\" as the program to run and providing a path to the model weights with the `ckpt_file` variable:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "id": "3c5fa39f-8798-4e41-8e2a-3a70a6be3906",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "2024-09-18 08:18:16,102 - INFO - --- input summary of monai.bundle.scripts.run ---\n",
      "2024-09-18 08:18:16,102 - INFO - > config_file: ['./MedNISTClassifier/configs/common.yaml',\n",
      " './MedNISTClassifier/configs/evaluate.yaml']\n",
      "2024-09-18 08:18:16,102 - INFO - > meta_file: './MedNISTClassifier/configs/metadata.json'\n",
      "2024-09-18 08:18:16,102 - INFO - > run_id: 'evaluate'\n",
      "2024-09-18 08:18:16,102 - INFO - > ckpt_file: './MedNISTClassifier/models/model.pt'\n",
      "2024-09-18 08:18:16,102 - INFO - ---\n",
      "\n",
      "\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Prediction: AbdomenCT. Ground-truth: AbdomenCT\n",
      "Prediction: BreastMRI. Ground-truth: BreastMRI\n",
      "Prediction: ChestCT. Ground-truth: ChestCT\n",
      "Prediction: CXR. Ground-truth: CXR\n",
      "Prediction: Hand. Ground-truth: Hand\n",
      "Prediction: HeadCT. Ground-truth: HeadCT\n",
      "Prediction: HeadCT. Ground-truth: HeadCT\n",
      "Prediction: CXR. Ground-truth: CXR\n",
      "Prediction: ChestCT. Ground-truth: ChestCT\n",
      "Prediction: BreastMRI. Ground-truth: BreastMRI\n"
     ]
    }
   ],
   "source": [
    "%%bash\n",
    "\n",
    "BUNDLE=\"./MedNISTClassifier\"\n",
    "export PYTHONPATH=\"$BUNDLE\"\n",
    "\n",
    "python -m monai.bundle run evaluate \\\n",
    "    --meta_file \"$BUNDLE/configs/metadata.json\" \\\n",
    "    --config_file \"['$BUNDLE/configs/common.yaml','$BUNDLE/configs/evaluate.yaml']\" \\\n",
    "    --ckpt_file \"$BUNDLE/models/model.pt\""
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6fd62905-4ea8-4f08-bcea-823074fc4ce4",
   "metadata": {},
   "source": [
    "## Summary and Next\n",
    "\n",
    "This tutorial has covered:\n",
    "* Creating full training scripts in bundles\n",
    "* Training a network then evaluating it's performance with scripts\n",
    "\n",
    "That's it to creating a bundle to match an existing script. It was mentioned in a number of places that best practice wasn't followed to stick to the original script's structure, so further tutorials will cover this in greater detail. "
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.12"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}
